Because of the widespread popularity of computers, most people have become comfortable with conventional computer input devices such as keyboards and pointing devices. The keystrokes on a keyboard, and the cursor movement and control provided by mice, trackballs, and joysticks are sufficiently intuitive to provide satisfactory interfaces for most computer-related tasks. Nonetheless, as computers become increasingly more indispensable, limits of a human-machine interface that depends upon pressing buttons and dragging a pointer with a mouse or other device tends to restrict how quickly and naturally computers can be used.
In seeking to further enhance the human-machine interface, ever-improving hardware capabilities have made possible systems that obviate the need to enter text with a keyboard. Personal digital assistants and tablet PCs can now recognize handwriting. Speech recognition software also enables users to operate computers and enter text by simply speaking into a microphone. Such systems can thus provide a more efficient and satisfying experience for users who prefer not to type on a keyboard or are less proficient in doing so, or who don't want to manipulate a pointing device.
As computers become even more commonplace throughout our environment, the desire to make computers and their interfaces even more user-friendly continues to promote development in this area. For example, the MIT Media Lab, as reported by Brygg Ullmer and Hiroshi Ishii in “The metaDESK: Models and Prototypes for Tangible User Interfaces,” Proceedings of UIST 10/1997:14 17,” has developed another form of “keyboardless” human machine interface. The metaDESK includes a generally planar graphical surface that not only displays computing system text and graphic output, but also receives user input by responding to an object placed against the graphical surface. The combined object responsive and display capability of the graphical surface of the metaDESK is facilitated using infrared (IR) lamps, an IR camera, a video camera, a video projector, and mirrors disposed beneath the surface of the metaDESK. The mirrors reflect the graphical image projected by the projector onto the underside of the graphical display surface to provide images that are visible to a user from above the graphical display surface. The IR camera can detect IR reflections from the undersurface of an object placed on the graphical surface, to detect the object and its disposition.
Others have been developing similar keyboardless interfaces. For example, papers published by Jun Rekimoto of the Sony Computer Science Laboratory, Inc. and associates describe a “HoloWall” and a “HoloTable” that display images on a surface and use IR light to detect objects positioned adjacent to the surface.
By detecting a specially configured object or by detecting IR reflected light from an object disposed on a graphical display surface, the metaDESK can respond to the contemporaneous placement and movement of the object on the display surface to carry out a predefined function, such as displaying and moving a map of the MIT campus. Thus, computing systems such as the HoloWall and metaDESK may provide a more natural degree of human machine interaction by providing the means for a computer to respond to specific objects.
By facilitating a more natural input arising from the person's interaction with a graphical display, such technologies not only broaden the manner in which a user might provide tactile input to a computing system, but also generally extend the ability of a user to provide input to a computing system. In conventional computing systems, a user provides only a single input to the computing system at one time with a pointing device or a keyboard. Using a graphical user interface, for example, a user might be able to select a number of documents to move into a folder or attach to an e-mail at a single time. But first, the user must select those items, and then associate them with the folder or e-mail by dragging or performing one or more mouse clicks. Single commands are made, sequentially mirroring the interaction of the human operator: with one pointing device, a user can perform one action at a time, therefore, modes of computing system input are designed to receive only one input at a time.
There is a degree of comfort with serial, single-input systems. When input is provided serially, a user is compelled by the serial input paradigm to perform actions in a logical, or at least a mandated sequence. In the example described above, to add a document as an attachment to an e-mail, the user first would identify the document to be attached, and then perform a second input to actually attach it to the e-mail. The serial input structure's limit of eliciting one input at a time inherently simplifies the process for the user so that the user does not become confused as to the nature or consequences of the actions being performed.
By contrast, an interactive display surface potentially enables a user—or multiple users—to simultaneously enter multiple inputs to the computing system. For example, a user might perform input by using both hands at the same time, or by positioning multiple physical objects adjacent to the interactive display surface. Examples of other methods and systems for detecting and responding to multiple objects placed on or near a graphical surface at one time are described in connection with other inventions owned by the assignee of the present application. These inventions are described in co-pending U.S. patent application Ser. No. 10/879,872, entitled “Using Size And Shape Of A Physical Object To Manipulate Output In An Interactive Display Application,” filed on Jun. 28, 2004, and application Ser. No. 10/883,515, entitled “Using Physical Objects To Adjust Attributes Of An Interactive Display Application,” filed on Jun. 30, 2004.
A possible concern with enabling one or more users to enter simultaneous multiple inputs to a computing system is the need to provide an indication to a user of choices made and the functions thus initiated. In other contexts where a person operates a device by providing multiple simultaneous inputs, it is common to receive visual, audible, or tactile feedback that indicates the simultaneous inputs are correct. For example, when a person playing a piano presses the intended keys at the same time, the resulting sound will indicate that the actions were correct. Similarly, a user operating an automobile with a manual transmission who fails to depress the clutch fully when shifting gears will receive multiple forms of audible and tactile feedback indicating that the user has failed to properly shift the transmission.
The ability to provide multiple simultaneous user inputs to a computing system does not necessarily provide such an unambiguous confirmation and feedback. For example, if a user were permitted to delete files by simultaneously pressing a delete key and a key corresponding to the first letter of a file name, the user might unknowingly and/or unintentionally delete a number of files by failing to change one of the inputs in time to carry out the intended action or by providing the wrong input. A user might be able to check that the desired files were indeed deleted, by opening a “recycle bin” or “deleted items” folder to see that the intended files were deleted. The user may choose not to bother with the additional step of having to check the contents of such a folder, and thus inadvertently destroy files the user intended to keep. Alternatively, the user may slow the pace for providing inputs to the system to make sure that such errors do not occur. In any case, being able to make multiple inputs at one time may result in only a slight time savings compared to entering inputs one at a time, or if a mistake is made, making multiple incorrect simultaneous inputs may result in more time being consumed than carrying out the action with more traditional sequential inputs.
It would therefore be desirable to provide confirmation to a user of actions that initiate specific functions. In particular, it would be desirable to provide such confirmation with an interactive display system or other computing system configured to permit multiple, simultaneous inputs, to enable a user to interact with the system more quickly and naturally. Ideally, the user should be provided such indicia without performing extra steps or having to slow the interaction to avoid mistakes that might otherwise arise when using a system able to simultaneously process multiple user inputs.